Text Based Knowledge Discovery with Information Flow Analysis

Information explosion has led to diminishing awareness: disciplines are becoming increasingly specialized; individuals and groups are becoming ever more insular. This paper considers how awareness can be enhanced via text-based knowledge discovery. Knowledge representation is motivated from a socio-cognitive perspective. Concepts are represented as vectors in a high dimensional semantic space automatically derived from a text corpus. Information flow computation between vectors is proposed as a means of discovering implicit associations between concepts. The potential of information flow analysis in text based knowledge discovery has been demonstrated by two case studies: literature-based scientific discovery by attempting to simulate Swanson’s Raynaud-fish oil discovery in medical texts; and automatic category derivation from document titles. There is some justification to believe that the techniques create awareness of new knowledge.